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ABSTRACT 


The Kauffman School is a public charter school that serves students from low-income 
neighborhoods in Kansas City, Missouri. This paper used a matched comparison group design to 
estimate the impacts of the Kauffman School on student achievement, attendance, and 
suspensions. We found that the Kauffman School had positive and statistically significant 
impacts on student achievement in mathematics, reading, and science. This paper also used 
surveys, interviews, focus groups, and classroom observations to describe the hallmarks and 
operations of the Kauffman School and explore possible mechanisms for its effects, informing 
the literature on school effectiveness. We found evidence that the Kauffman School’s hallmarks 
are largely being implemented faithfully, and that key stakeholders believe the Kauffman 
School’s methods are having a positive influence on students’ behavior, attitudes, and 
performance. 


The authors are grateful to Melissa Bardwell, Leigh Ann Grant-Engle, Timothy Wittmann, and 
the Missouri Department of Elementary and Secondary Education for providing the data used in 
this report. We would like to thank the following Mathematica staff who contributed to the 
study: Brian Gill provided important feedback and comments on the report and Ira Nichols- 
Barrer gave valuable input on the study design. Cindy George and Jennifer Littel provided 
editing services and Jessica Hegedus formatted the report. The report is based on findings from 
an evaluation funded by the Ewing Marion Kauffman Foundation. 
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I. INTRODUCTION 


Charter schools represent a rapidly growing part of the American public education system. 
They are intended to serve as laboratories for innovation, and the growth of the sector provides 
increased opportunities to leam from their experience. Nearly 3 million students are currently 
served by 6,700 charter schools nationwide, which is an increase from 2,800 charter schools and 

0. 7. million students in 2003 (National Alliance for Public Charter Schools 2016). With the rapid 
growth of charter schools, stakeholders’ interest has grown in both the academic impact of 
charter schools and also in charter school policies and practices which may contribute to their 
impacts on student achievement growth. What follows is a study of the effectiveness, policies, 
and practices of the Ewing Marion Kauffman School, a charter school in Kansas City, Missouri. 

Background about the Kauffman School 

The Kauffman School was opened by the Ewing Marion Kauffman Foundation in fall 2011, 
initially serving 5th grade, and expanding upward one grade each year so that it will ultimately 
serve grades 5 through 12. For many years, the Kauffman Foundation has focused efforts on 
improving education for children in Kansas City. Before opening the Kauffman School, the 
Kauffman Foundation operated several programs that addressed the challenges faced in urban 
education, such as Project Early (an early childhood program), Project Choice (a high school 
dropout prevention program), and the Kauffman Scholars program (an after-school college 
preparatory program). The success of these programs led Foundation leaders to consider the 
impact they might have on students in Kansas City if they established a charter school. In March 
2009, the Kauffman Foundation established its school design team. The school design team 
assessed Kansas City’s low-income families’ educational needs and identified the best practices 
of successful charter schools from across the country to inform the Kauffman School’s policies. 
The key hallmarks of the Kauffman School include: 

1 . High attendance and behavioral expectations. Kauffman students are expected to 
maintain at least a 95 percent average daily attendance rate and to fully comply with all 
school policies and procedures. 

2. Ambitious academic goals. The Kauffman School expects its students to excel 
academically and achieve at least 1 .25 years of growth in mathematics, science, and reading 
each year. These goals are regularly discussed by school administrators and staff, teachers, 
students, and parents. 

3. Extended school day and year. As part of the effort to meet the academic growth target, 
the Kauffman School operates an extended school day and year. The Kauffman School 
estimates that students receive approximately five weeks of additional instruction per year 
relative to other public school students in Kansas City. 

4. Increased mathematics and reading instructional time. Kauffman students have two 
periods of mathematics instruction each day, one period of reading instruction, and one 
period of literature/writing instruction. In addition, starting in the Kauffman School’s second 
year of operation, all students attended an instructional support class each day in which they 
received tutoring and special instruction. Struggling students received additional instruction 
and practice in any subjects in which they needed help, and high-performing students 
received advanced instruction. 
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5. Intensive, data-driven decision making. With its strong emphasis on results, the Kauffman 
School employs a large assessment portfolio so that teachers and administrators can make 
data-driven decisions about how to adapt instruction to best meet students’ needs. In 
addition to teacher-developed “exit tickets,” 1 quizzes, and tests to measure understanding 
and academic progress, the Kauffman School’s assessment portfolio includes the following: 

Achievement Network (ANet) assessments in mathematics and reading, revised by 
Kauffman School teachers to be consistent with Missouri State Standards, administered 
every six weeks, 

Strategic Teaching and Evaluation of Progress (STEP) assessments to measure students’ 
reading growth, administered six times per year, 

Northwest Evaluation Association (NWEA) assessments in mathematics, reading, and 
science, administered twice a year, and 

Missouri Assessment Program (MAP) standardized tests in mathematics, reading, and 
science, administered annually by the State of Missouri. 

6. Extensive teacher professional development. The Kauffman School places a significant 
emphasis on teachers’ professional development, with teachers experiencing (1) frequent 
classroom observations and feedback; (2) weekly one-on-one meetings with instructional 
coaches; and (3) group-based professional development sessions every Friday afternoon, 
focused on various topics related to curriculum, instruction, assessment, student behavior, 
and school culture. 

7. Well-established cultural norms. The Kauffman School takes an intentional approach to 
establishing a culture that consists of shared values, expectations, and norms, epitomized by 
the Kauffman SchooFs motto: “Creating College Graduates.” Continuous efforts are made 
to explicitly communicate the values, expectations, and norms to all school staff, students, 
and families. 

Taken together, these features of the Kauffman School are similar to those found in “no 
excuses” charter schools, yet the Kauffman School does not identify itself as a “no excuses” 
school. When Kauffman School leaders think about how the tenn “no excuses” applies to their 
school, they describe a learning environment in which every adult holds high academic and 
behavioral expectations of all students and the adults do not make excuses for themselves if 
students are not meeting expectations. 

Study summary 

This paper uses a quasi-experimental, matched comparison group design to estimate the 
impacts of the Kauffman School on student achievement, attendance, and suspensions. 

According to our findings, during its first three years of operation, the Kauffman School had 
positive, statistically significant, and educationally meaningful impacts on student achievement 
in mathematics, reading, and science. 


1 Exit tickets are short questions or tasks that students complete at the end of the class period. These enable teachers 
to track the progress of their students’ understanding of the course material on a regular basis. 
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The estimated impact of Kauffman School on student attendance is positive and significant 
in some grades and not significantly different from zero in others. We find that the Kauffman 
School suspends students at a higher rate than other schools in Kansas City. However, we also 
see that the number of days suspended for Kauffman students is on average small relative to the 
additional instructional time afforded by the Kauffman School’s extended school day and year. It 
appears then that the discipline policy at the Kauffman School can be strictly enforced without 
resulting in a substantial loss of learning time for suspended students. 

Data from surveys (of teachers, students, and parents), interviews, and classroom 
observations help us explore the mechanisms that might have contributed to the large positive 
impacts on student achievement and the high suspension rates. By combining information on 
school policies and operations with data on student impacts, the study aims to enhance our 
understanding about how schools can ensure that students develop the skills needed for academic 
success. Our study also contributes to the current policy debate about disparities in suspension 
rates and student achievement by placing the high suspension rates at the Kauffman School in 
context with the Kauffman School’s large positive impacts on student achievement. 

II. PREVIOUS LITERATURE 


Large nationwide studies of charter schools have shown mixed impacts on student 
achievement on standardized tests. Gleason et al. (2010) analyzed a sample of oversubscribed 
charter middle schools with lottery admission processes and found, on average, no detectable 
effects on student achievement, although a subsample of these schools serving low-income 
students showed significantly positive impacts on students’ mathematics achievement. Across 
the 27 states included in the Center for Research on Education Outcomes (CREDO) analysis of 
charter schools, there was no average impact on students’ mathematics achievement and a small 
positive impact on students’ reading achievement (CREDO 2013). A national study of charter 
school management organizations (CMOs) found that, on average, CMOs had no detectable 
effects (Furgeson et al. 2012). Berends et al. (2010) conducted a study of the impacts of 62 
charter schools in Idaho, Indiana, and Minnesota and found no statistically significant effects. 
Zimmer et al. (2012) found mixed results in their study of charter schools and traditional public 
schools in two states — Ohio and Texas — and five large school districts — Chicago, Denver, 
Milwaukee, Philadelphia, and San Diego. Sass (2006) used data covering all public school 
students in Florida and found that, on average, charter schools in their first few years of 
operation had negative impacts on student achievement, but charter schools in operation for five 
or more years have positive and significant impacts on student achievement. 

Many studies of charter schools in urban settings, and especially among schools serving 
poor and minority students, like the population served by the Kauffman School, have shown 
significant positive impacts on student achievement. Urban charter schools in 4 1 regions across 
the United States were shown to have on average positive and significant impacts on student 
mathematics and reading achievement (CREDO 2015). Abdulkadiroglu et al. (2009) found large 
positive impacts on student achievement of attending a charter school in Boston relative to 
traditional public schools, particularly among oversubscribed charter schools with admission 
lotteries. The New York City charter schools analyzed in Hoxby et al. (2009) were also shown to 
have substantial positive effects on student achievement. Dobbie and Fryer (2011) showed large 
positive impacts on student achievement for two Promise Academy charter schools in the 


5 



WORKING PAPER 43 


MATHEMATICA POLICY RESEARCH 


Harlem Children’s Zone in New York City. Angrist et al. (2010) showed large positive impacts 
on student achievement for KIPP Lynn, which is part of the Knowledge Is Power Program 
(KIPP) network of charter schools that serve predominately low-income students. In a 
comprehensive study of KIPP middle schools, Tuttle et al. (2013; 2015) also found substantial 
positive average impacts on student achievement. Witte et al. (2007) showed that charter schools 
in Wisconsin performed better than traditional schools mainly at improving the achievement of 
low-perfonning students who are initially at the minimal or below basic levels. Grigg and 
Bonnan (2014) found no impacts for students attending two oversubscribed charter elementary 
schools in Colorado, although they found suggestive evidence of positive impacts on 
achievement for non- white students. 

In addition to academic perfonnance, the impact of charter schools on disciplinary 
outcomes, such as attendance and suspension rates, has gained attention by educators and 
stakeholders. However, this relationship has been less widely studied. Dobbie and Fryer (2011) 
found that children attending both the Promise Academy middle and elementary school had 
significantly fewer absences than comparison students. Angrist et al. (2011) estimated that urban 
charter middle school attendance was associated with a higher number of days suspended 
compared to urban traditional public schools. However, Gleason et al. (2010) found that students 
who attended oversubscribed charter schools in their study experienced a similar number of 
school absences and were equally likely to be suspended during the school year compared to 
students who also applied, but were not admitted to these schools. 

Our paper adds to this growing body of research related to the academic and behavioral 
impacts of charter school attendance by examining the hallmarks of the Kauffman School and 
how they are implemented to understand the factors contributing to the observed impacts. Below 
we discuss the literature on the effectiveness of school policies and programs related to each of 
the Kauffman School’s key hallmarks. 

High expectations for behavior and attendance 

High expectations for behavior and attendance is one of the fundamental principles of many 
charter schools, particularly of schools that believe in a “no excuses” approach to education. “No 
excuses” schools combine intensive classroom and learning policies geared toward the 
realization of high academic goals (Angrist et al. 2011). Such policies involve the consistent 
enforcement of discipline and behavior codes that reward students who consistently behave well, 
and impose predictable and immediate sanctions on those who violate rules (Thernstrom and 
Themstrom 2004). 

Studies have also shown that charter schools vary substantially in both the types of 
behavioral policies they implement and the level of consistency with which they enforce them. 
Studies indicate that schools with more comprehensive and consistently enforced schoolwide 
behavior systems have larger impacts on student achievement in mathematics and reading 
(Dobbie and Fryer 2013; Furgeson et al. 2012; Knechtel et al. 2015; Tuttle et al. 2013). 

Specific disciplinary tactics, such as the removal of disruptive or misbehaving students from 
the classroom, may directly affect academic outcomes. It is possible that “zero tolerance” 
policies positively affect student learning by removing disruptive students from school and 
deterring misbehavior (Ewing 2000; Noguera 1995). However, schools’ use of suspension has 
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also been linked to short-term and long-term academic delays for students (Arcia 2006; Rausch 
and Skiba 2005). These academic delays may be a result of suspensions reducing academic 
learning time or opportunity to leam (Advancement Project/Civil Rights Project 2000). There 
has also been heightened awareness of suspension rates of minority students, with studies linking 
the racial achievement gap to the gap in school discipline rates (Losen et al. 2015; Morris and 
Perry 2016). Nonetheless, because suspension rates are driven by school policy and student 
behavior, and often are related to prior achievement (Rausch and Skiba 2005), it is difficult to 
identify precisely how these elements contribute to differences in achievement gains or losses. 

School attendance is another behavioral expectation many charter schools emphasize and 
enforce with fonnal policies. For example, Hoxby et al. (2009) report that about half of New 
York City’s charter school students attended a school where parents were asked to sign a 
contract that included established expectations about attendance and on-time arrival at school. 
There is also empirical evidence documenting a negative relationship between absenteeism and 
short- and long-tenn academic outcomes (Buehler et al. 2012; Spradlin et al. 2012; Chang and 
Romero 2008). 

Ambitious academic goals 

Charter schools that hold high expectations for student achievement have been shown to 
produce larger academic gains in core subjects compared to charter schools that lack explicit 
policies related to high expectations (Dobbie and Fryer 2013) and compared to traditional public 
schools (Berends et al. 2010; Goldring and Cravens 2007). In an analysis of 76 charter schools in 
Colorado, Idaho, Indiana, and Minnesota, Goldring and Cravens (2007) found that, compared to 
traditional public schools, charter schools are more likely to implement in-school organizational 
conditions that support teachers’ efforts to improve instruction and focus on achievement, such 
as principal leadership, professional learning communities, and teacher decision making 
authority. Berends et al. (2010) extended this research using a subsample of elementary schools 
and found that a school’s “focus on achievement” level was significantly and positively 
associated with gains in mathematics achievement. In their study on the inner workings of 35 
charter schools in New York City, Dobbie and Fryer (2013) found that prioritization of high 
academic and behavioral expectations was associated with higher annual mathematics and ELA 
achievement gains for charter school students. 

As noted above, expectations and ambitious academic goals are fundamental principles of 
“no excuses” schools. “No excuses” schools place a strong emphasis on educational practices 
including frequent testing, increased instructional time, and a relentless focus on mathematics 
and reading achievement (Dobbie and Fryer 2013). The “no excuses” teaching and learning 
philosophy has been associated with large academic gains (Angrist et al. 2011; Dobbie and Fryer 
2013). However, in Dobbie and Fryer (2011), the “no excuses” designation became statistically 
insignificant when controlling statistically for more instructional time, more effective teachers, 
the use of data and high-dosage tutoring, and high expectations, suggesting these policies are 
more important predictors of school effectiveness than a school’s overarching “no excuses” 
philosophy. 
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Extended school day and year 

Extended school days and years are intended to provide students with more instructional 
time to help attain greater academic gains. Schools extend classroom time not only to help 
students who are performing below grade level to catch up, but also to maintain a momentum of 
instruction that will help students meet high academic goals. Farbman and Kaplan (2005), for 
instance, suggest more time in school can advance learning when it affords students 
opportunities to spend more time on task, delve deeply into subject matter, engage in 
experimental learning, and receive tutoring and help with homework. 

Relative to district schools, charter schools tend to have significantly longer school days and 
slightly longer school years (Farbman and Kaplan, 2005; Furgeson et al, 2012; Gleason et al., 
2010). For example, oversubscribed charter middle schools studied by Gleason et al. (2010) had 
significantly longer school days (7.2 versus 6.7 hours) and marginally longer school years (181.2 
versus 179.9 days) than comparison schools. In addition, CMO principals studied by Furgeson et 
al. (2012) reported that their schools provide an additional 134 hours of instruction annually, 
compared to hours of instruction reported by traditional district school principals. While Zimmer 
and Buddin (2007) and Knechtel et al. (2015) found time in school had no effect on achievement, 
others have found a positive association with an increase in instructional days and student gains 
in mathematics and reading (Furgeson et al. 2012; Hoxby and Murarka 2009). 

Increased time for mathematics and reading instruction 

A more important predictor of academic outcomes than increased instructional time may be 
how instructional time is spent. Dobbie and Fryer (2011) found that, after controlling for 
operational policies, schools that add more instructional time for mathematics have higher annual 
achievement gains in that subject. Tuttle et al. (2013) found that average impacts on reading and 
mathematics achievement are larger in KIPP schools that spend relatively more time on core 
academic activities, a factor not necessarily correlated with length of school day. However, more 
recent data on KIPP schools show no relationship between time in core subjects and impacts on 
academic achievement (Knechtel et al. 2015). 

Zimmer and Buddin (2007) also found that spending more hours on mathematics was 
positively associated with higher mathematics and reading scores in both charter and traditional 
public elementary schools, but increased reading/language arts instruction in elementary schools 
was not significantly correlated with higher test scores. The amount of time middle and high 
schools spend on mathematics and reading also was found to have little effect on test scores in 
charter and traditional public schools. 

Intensive data-driven decision making 

Teachers and administrators often use assessment data to track student progress and inform 
decisions about instruction such as adjusting tutoring groups and assigning remediation, 
modifying instructional material, and developing individualized student goals. Although 
Knechtel et al. (2015) found that KIPP teachers who receive more frequent support in the use of 
data to drive instruction have smaller positive impacts on reading, other studies have found that 
strategic use of data to inform decisions is positively related to achievement in charter schools 
(Dobbie and Fryer 2011; Furgeson et al. 2012). For example, reviewing student test results has 
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been li nk ed to modest academic outcomes, whereas frequently revising teaching plans in 
response to assessment data results is positively and significantly predictive of achievement 
(Furgeson et al. 2012). Likewise, Dobbie and Fryer (2011) found charter schools that give five or 
more interim assessments during the school year and that implement four or more differentiation 
strategies have higher gains in mathematics and ELA achievement than schools that conduct 
fewer assessments and implement fewer differentiation strategies. These findings suggest that 
student assessment data may be useful only to the extent that they are associated with decision 
making and actual changes in instructional practices (Furgeson et al. 2012). 

Extensive teacher professional development 

Schools may aim to increase teacher effectiveness by offering professional development 
opportunities such as classroom observation and feedback, peer coaching, meetings with school 
leaders, and in-service workshops. Both charter schools and traditional public schools offer 
frequent professional development, yet the types of opportunities emphasized tend to vary across 
schools. (Zimmer and Buddin 2007; Wei et al. 2014). For example, charter schools in California 
studied by Zimmer and Buddin (2007) placed a greater emphasis on mentoring and coaching 
than traditional schools. Evidence suggests that CMOs are more likely to practice more frequent 
and intense professional development than district schools. Principals of CMO schools studied 
by Furgeson et al. (2012) reported providing more observations of and feedback to new teachers 
and required that these teachers submit lesson plans for review more often than did district 
school principals. 

Although there is some evidence that professional development opportunities in both 
traditional public and charter schools are largely unrelated to reading and mathematics 
achievement (Zimmer and Buddin 2007), some studies have found that placing an emphasis on 
coaching new teachers and conducting more one-on-one meetings translates into larger academic 
gains (Dobbie and Fryer 2011; Furgeson et al. 2012). Dobbie and Fryer (2011), for example, 
found schools that give formal or infonnal feedback 10 or more times per semester had 
significantly higher annual mathematics and ELA gains than other schools. Furgeson et al. 

(2012) also found that the extent to which teachers are observed is correlated with the frequency 
with which they modify their lesson plans using student assessments, which is, in turn, 
associated with positive impacts on student achievement. 

Well-established cultural norms 

In addition to investing in the professional development of its teachers and staff, schools 
often establish comprehensive policies that promote a school culture aimed to develop the 
character of their students while improving student capacity for learning. Using interviews, focus 
groups, and class observations from eight schools in the northeast, Farbman and Kaplan (2005) 
concluded a positive school culture is a key feature of what helps a school reach its goals and 
produce strong student outcomes. Research suggests policies that aim to establish shared norms 
and values may be more common among CMOs and charter schools than traditional public 
schools. Goldring and Cravens (2007), for instance found that, compared to traditional public 
schools, charter schools scored higher on a composite measure of instructional focus, level of 
expectations, school climate, monitoring of student progress, and school-community relations. 
Lake et al. (2010; 2012) found in their comprehensive evaluation of 43 CMOs that nearly all 
CMO leaders required their schools to promote certain shared beliefs and values. 
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III. METHODOLOGY AND DATA FOR IMPACT ANALYSIS 


Methodology 

Comparing Kauffman students to students from other Kansas City schools 

Because all Kauffman students have chosen to enroll in the Kauffman School, they might 
differ from other Kansas City students in important ways. Measuring the effect of the Kauffman 
School on student achievement requires identifying a comparison group of Kansas City students 
who, as of 4th grade (before the Kauffman School’s 5th-grade entry year), are similar to the 
students who are about to enter the Kauffman School. Otherwise, any difference we find in later 
student outcomes might not be due to the effect of the Kauffman School. 

To guarantee that the comparison group is similar, the gold standard research design would 
involve conducting a lottery wherein some of the students who apply to the Kauffman School are 
randomly selected to attend the Kauffman School and others are randomly denied admittance. 

The achievement of these two randomly established groups could then be fairly compared (based 
on the assumption that any naturally occurring differences among students would be randomly 
distributed between the two groups). However, the Kauffman School was not sufficiently 
oversubscribed during its first three years of operation to use this approach. Instead, we 
employed the next best approach: using data from students across Kansas City to identify a 
matched comparison group of students who were similar to Kauffman School students when they 
were in 4th grade. 

To construct a comparison group of students, we implemented a propensity-score matching 
procedure. Students attending other schools in Kansas City were matched to Kauffman School 
students based on characteristics such as prior test scores, prior attendance, prior suspensions, 
and demographic characteristics. This approach is commonly used as an alternative to random 
assignment to evaluate the impacts of charter schools and has been shown to produce valid 
impact estimates that replicate the results of experimental research designs (Tuttle et al. 2013; 

Gill etal. 201 5). 2 

Constituting the Kauffman student group 

Throughout our analysis, we classify any student who was enrolled for at least part of a year 
in the Kauffman School as a Kauffman student. Classifying students in this manner defuses the 
potential criticism that the Kauffman School’s effects are overestimated because low-achieving 
students have left the charter school. However, including these students might lead to 
understating the impact of the Kauffman School on student achievement because students who 
left the Kauffman School early in the school year would have received very little influence from 
the Kauffman School. Similarly, students from the Kauffman School’s first cohort who left after 
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~ See Section IV of the appendix for more details about the implementation of the propensity-score matching 
procedure. In Johnson et al. (2016), we examine the sensitivity of the results to the use of all students in Kansas City 
public schools as the comparison group rather than those selected by the propensity-score matching procedure and 
find similar results. 
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5th grade are kept in the treatment group for the 6th and 7th grade analysis samples. 3 Again, this 
is a conservative analytic approach that reduces the risk of overestimating the impact, but it 
means that the full impact on students who continue in the Kauffman School for two or three 
years is likely to be underestimated. 4 

Data for our analysis were available for three cohorts of Kauffman students. Cohort I 
students are those who entered the Kauffman School as 5th graders in 2011-2012 (the year the 
Kauffman School opened). Cohort II students are those who entered the Kauffman School as 5th 
graders in 2012-2013. Cohort III students are those who entered the Kauffman School as 5th 
graders in 2013-2014. 

In the next section, we present three-year, two-year, and one-year impact estimates for the 
Kauffman School. The three-year impacts are based on the outcomes of cohort I students, who 
were 7th graders in 2013-2014. 5 The two-year impacts are the average of the Kauffman School’s 
estimated impact on cohort I, the 6th graders in 2012-2013, and its impact on cohort II, the 6th 
graders in 2013-2014. The one-year impacts are the average of the Kauffman School’s estimated 
impact on all three cohorts in their 5th-grade year. 

Data and descriptive statistics 

In this section, we provide details about the data used in our main analysis of the impact of 
the Kauffman School on student outcomes. We also present a set of descriptive statistics to show 
how Kauffman students compare to students in other schools in the city, in terms of prior 
achievement and demographic characteristics. 

Data characteristics 

The data used in our main analyses were provided by the Missouri Department of 
Elementary and Secondary Education. The data consisted of MAP test scores in mathematics, 
reading, and science, along with attendance and suspension data for all students in Missouri who 
were in the 5th, 6th, or 7th grade in the 2011-2012 through 2013-2014 school years. 6 We also 
obtained for these students data on their prior (3rd and 4th grade) test scores, prior attendance, 
prior suspensions, and demographic characteristics. We limited our potential comparison group 
to students attending schools in the borders served by the Kansas City Public Schools district 
(KCPS). (See Section III of the appendix for details about the analysis sample selection process.) 


3 

Only students who entered the Kauffman School in 5th grade are included in the treatment group for this analysis. 
During the second and third year of operation, the Kauffman School did not admit new 6th or 7th grade students to 
the School. 

4 In Johnson et al. (2016), we discuss the issue of attrition in more detail and present attrition-adjusted impact 
estimates that approximate the impact of the Kauffman School for students who remain enrolled in the School. 

5 Cohort I students who repeated 5th grade in 2012-2013 or 6th grade in 2013-2014 are also included in the 
Kauffman student group for the three-year impact calculations. See Johnson et al. (2016) for details about how grade 
repeaters are treated in the analysis. 

6 The official name of the reading MAP assessment is “Communication Arts.” 
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The results in this report are based on a matched comparison group selected from all 
students attending schools within the borders of KCPS, including other charter schools. This 
group of students is likely the most relevant for our analysis; using them to construct the 
comparison group provides an impact estimate that indicates how much more or less a Kansas 
City student would be expected to achieve if that student were to enroll in the Kauffman School 
rather than a typical Kansas City school. Our main results include students at both traditional 
public schools and charter schools in Kansas City in the comparison group; see Johnson et al. 
(2016) for results reported separately for each comparison group of schools. 

Data on one or more of the variables used as baseline controls are missing for many 
students. About 15 percent of the students we could potentially include in our analysis sample in 
the most recent year of data are missing data on one or more of the baseline control variables. To 
avoid dropping them from the analysis, we employed a multiple imputation procedure to 
estimate their missing baseline values. 7 We also analyzed the data without using imputation and 
found similar results (see Johnson et al. 2016). 

Descriptive statistics: What types of students attend the Kauffman School? 

Full descriptive statistics for the first three cohorts of Kauffman students, compared with 
students in other Kansas City public schools, are presented in Table A. 2 in the appendix. On 
average, Kauffman students had 4th-grade MAP scores that were below the statewide average in 
both mathematics and reading. Students at the Kauffman School were also predominately from 
low-income and minority families: across the first three cohorts, at least 80 percent of Kauffman 
students were eligible for free or reduced-price lunches and at least 83 percent were black or 
Hispanic. 

Although, on average, the Kauffman students had performed below the state average on the 
4th-grade MAP (before enrolling in the Kauffman School), they had higher 4th-grade MAP test 
scores than other students in Kansas City public schools and were less likely to receive any 
special accommodations while taking the 4th-grade MAP test. Compared to other students in 
Kansas City public schools, Kauffman students also were more likely to be black, were less 
likely to be Hispanic, and had slightly higher average 4th-grade attendance rates. Kauffman 
students were generally similar to other Kansas City public school students with respect to free 
or reduced-price lunch and disability rates, although there were some significant differences 
across cohorts. 

As previously mentioned, we used a matching procedure to identify a group of comparison 
students who were similar to Kauffman students with respect to baseline characteristics. Table 
A. 5 in the appendix provides descriptive statistics for the matched comparison groups relative to 
each cohort of Kauffman students. By design, there are no significant differences in baseline 
characteristics between the Kauffman and matched comparison groups. Some Kauffman students 
are not included in the main analyses because we were unable to find matching comparison 
group students. For instance, when creating a comparison group similar to all Kansas City public 
school students in 2013-2014, we were unable to find a match for 1 1 percent of Kauffman 
students. See Table A. 4 in the appendix for details on the number of Kauffman students who 


7 


See Section III of the appendix for more details about our imputation procedure. 
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were excluded for this reason. As a robustness check, we perfonned an analysis that did not use 
matching but relied exclusively on a regression model to account for baseline differences 
between Kauffman and comparison students. This analysis included all Kauffman and 
comparison students with non-missing data; the results were similar to the main findings 
(Johnson et al. 2016). 

IV. THE IMPACT OF THE KAUFFMAN SCHOOL ON STUDENT ACHIEVEMENT 


In this section, we present the main impact estimates for the Kauffman School on student 
achievement as measured by the MAP exams in mathematics, reading, and science. We present 
three- and two-year impact estimates for mathematics and reading, as well as one-year impact 
estimates for all three subjects. We also describe various ways of interpreting the impact 
estimates and place the size of the estimates in the context of findings for other evaluations of 
charter school effectiveness. 

Impacts on MAP test scores 

The impact estimates for the Kauffman School on student achievement in each MAP subject 
are displayed in Table 1. The results are based on linear regression models that include the 
Kauffman students and matched comparison students and control for small remaining differences 
in prior achievement and other baseline characteristics. 8 As noted previously, any student who is 
enrolled in the Kauffman School as a 5th grader for at least part of the school year is included in 
the Kauffman group for all impact estimates. The impact estimates should therefore be 
interpreted as the average effect of enrolling in the Kauffman School, accounting for the 
possibility that students may leave. The results are displayed in effect size units, which can be 
interpreted as how many student test score standard deviations higher or lower Kauffman 
students are perfonning relative to students in the comparison groups (after controlling for 
baseline achievement). 9 Standard errors are displayed in parentheses below each estimate, and 
asterisks indicate whether the estimate is significantly different from zero. 


o 

The covariates include all the variables summarized in Table A.2. We also include 3rd-grade mathematics and 
reading MAP scores, second- and third-order polynomial terms for 4th-grade MAP scores, and indicator variables 
that equal one if a student has imputed prior test scores or imputed attendance or suspension data. 

9 The statewide standard deviations of 7th-grade MAP scores were 41 in mathematics and 36 in CA. The statewide 
standard deviations of 6th-grade MAP scores were 40 in mathematics and 33 in CA. The statewide standard 
deviations of 5th-grade MAP scores were 43 in mathematics, 36 in CA, and 34 in science. 
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Table 1. Impact of Kauffman School on MAP test scores (citywide 
comparison group) 


Mathematics 

Reading 

Science 

Sample size 1 

One-year impact estimates 

0.12** 

(0.04) 

0.13“ 

(0.04) 

0.43“ 

(0.04) 

2,242 

Two-year impact estimates 

0.27** 

(0.05) 

0.19“ 

(0.06) 

n.a. 

1,181 

Three-year impact estimates 

0.57** 

(0.07) 

0.41“ 

(0.08) 

n.a. 

534 


Note: This table displays impact estimates in effect size units. The first row presents the average one-year impact 

estimates for cohort I, cohort II, and cohort III 5th graders. The second row presents the average two-year 
impact estimates for cohort I and cohort II 6th graders. The third row presents the three-year impact 
estimates of cohort I 7th graders (the only cohort that has completed three years in the school). Standard 
errors are displayed in parentheses below each impact estimate. The sample size represents the total 
number of Kauffman and matched comparison students entering each analysis. One asterisk (*) indicates 
impacts that are significantly different from zero at the 5 percent level. Two asterisks (**) indicate impacts 
that are significantly different from zero at the 1 percent level. 

n.a. = not applicable. 

The first row of Table 1 shows the amount of additional growth realized by Kauffman 
students relative to matched comparison students in all other Kansas City public schools one year 
after enrollment in the Kauffman School. These numbers represent the average effect size 
estimate for the first three cohorts of 5th-grade students. 10 The one-year impact estimates for the 
Kauffman School are positive and statistically significant in mathematics, reading, and science. 
Caution should be used when interpreting the science estimate, because no prior year science test 
score was available to use in the propensity-score matching procedure or as a control variable in 
the regressions. The statewide science assessment in Missouri is first administered in 5th grade, 
so the only baseline test score variables available for use in the analysis of 5th-grade science 
impacts are prior scores in mathematics and reading. 

The two-year impacts are statistically significant and larger than the one-year impact 
estimates in both subjects. The third row of Table 1 reports the effect size estimates three years 
after enrollment. The three-year impact of the Kauffman School on student MAP growth is 
especially large, amounting to 0.57 standard deviations in mathematics and 0.41 standard 
deviations in reading. Collectively, these results demonstrate that the impact of the Kauffman 
School accumulates for students who are enrolled for multiple years. 

Interpretation of Kauffman School impact estimates 

To support the interpretation of the effect size estimates, we converted the effect sizes into 
two alternative units: (1) years of learning growth and (2) the change in state test score percentile 
ra nk for the average Kauffman student. 


10 The effect sizes were estimated separately for each cohort of students. To calculate the impact estimates in Table 
1, we averaged these effect sizes together, weighting by the number of Kauffman students in the analysis sample for 
each cohort. 
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Results as years of learning growth 

We can translate the effect sizes presented in the previous section into an approximate 
measure of the years of additional learning growth experienced by Kauffman students using the 
results in Bloom et al. (2008). 11 Performing this conversion on the results from the main 
comparison group yields the results shown in Figure 1 . Three years after enrollment, Kauffman 
students experienced a cumulative additional 1.35 years of learning in mathematics and 1.29 
years in reading. Note that caution is warranted when interpreting the Kauffman School effect 
size results in terms of years of learning because the accuracy of these conversions depends on 
how similar achievement growth on the MAP exams is to the vertically scaled assessments 
analyzed in Bloom et al. (2008). 12 


Figure 1. Kauffman School estimates of additional years of learning growth 



1.35 


1.29 



One-year impacts Two-year impacts 

■ Mathematics ■ Reading Science 


Three-year impacts 


Note: The additional growth for all impact estimates is significantly different from zero. 


1 1 Using a set of widely administered vertically scaled assessments, Bloom et al. (2008) estimated that the typical 
5th-grade student grows 0.56 standard deviations in mathematics, 0.40 standard deviations in reading, and 0.40 
standard deviations in science. They also estimated that the typical 6th-grade student grows 0.41 standard deviations 
in mathematics and 0.32 standard deviations in reading, whereas a typical 7th-grade student grows 0.30 standard 
deviations in mathematics and 0.23 standard deviations in reading. To convert the one-year impact estimates of the 
Kauffman School into units of additional years of learning, we divided the impact estimates by the typical growth of 
5th-grade students in each subject. We used a similar method to convert the two- and three-year impact estimates 
into additional years of learning growth. For the two- and three-year results, we divided the impact estimates by the 
average of the typical growth across all grades included in each analysis. 

12 

" If typical achievement growth on the MAP is less than growth on the assessments analyzed in Bloom et al. 

(2008), then this conversion will underestimate the additional years of learning growth achieved by Kauffman 
students and vice versa. The scale of the MAP assessments is based, in part, on the Terra Nova exams, giving the 
MAP some of the characteristics of a vertically scaled exam. Thus, in principle, we could use average growth on the 
MAP in place of the numbers from Bloom et al. (2008). However, there are known issues with the MAP vertical 
scale when students show no growth on average between grades 5 and 6 (CTB McGraw-Hill 2012). Therefore, we 
did not attempt to use the vertical scale of the MAP to convert effect sizes into units of years of learning. 
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Results as test score percentile ranks 

As a second interpretation approach, we report the change in state test score percentile ranks 
that the average Kauffman student would expect to achieve as a result of attending the Kauffman 
School rather than an average Kansas City school. In 4th grade, before entering the Kauffman 
School, the average Kauffman student from the first cohort was at the 36th percentile in the state 
mathematics test score distribution and at the 39th percentile in the state reading test score 
distribution. 13 Three years after enrollment at the Kauffman School, the average student moved 
to the 58th percentile in mathematics and the 55th percentile in reading. 14 On average, students 
enrolled at the Kauffman School moved from substantially below average in the state 
distribution to above the state average three years after enrollment at the Kauffman School. 

Comparison of Kauffman School estimated impacts to charter schools 

nationwide 

In this section, we compare the three-year impact estimates of the Kauffman School to the 
average impacts from some of the other samples of charter schools described in Section II. The 
comparisons are summarized in Figure 2. The achievement growth experienced by students at 
the Kauffman School is substantially higher than the average growth experienced by charter 
school students nationwide. The Kauffman School also has larger impacts than the average 
Boston, New York City, and KIPP charter school. 13 It is important to note that Figure 2 displays 
the average impacts of the charter schools in these groups. Some individual schools or subsets of 
these groups have larger estimated impacts than the Kauffman School. For example, the three- 
year impact estimates for over-subscribed Boston charter middle schools analyzed in 
Abdulkadiroglu et al. (2009) are equivalent to 0.5 1 standard deviations in reading and 1.6 
standard deviations in mathematics. The three-year impact estimates for the Kauffman School 
are approximately double the size of the two-year impact estimates. When the Kauffman School 
two-year impact estimates are compared to those reported in other studies of charter schools, the 
estimates fall within the range of other highly successful charter programs but are still 
substantially larger than the impact estimates from national charter school studies (see Johnson et 
al. 2016). 


13 

These calculations are based on the current analysis sample of cohort I students only, because this is the sample 
used to calculate the three-year impact estimates. 

14 The percentile ranks three years after enrollment at the Kauffman School were calculated by taking the average 
4th-grade z-scores of Kauffman students and adding the three-year effect size estimates. These calculations assume 
that the percentile rank of the average student in Kansas City does not change over time. 

15 The three-year impact estimates reported in this section are generally obtained by tripling the average annual 
impact estimates reported by the authors. The exceptions to this are the KIPP study, the charter lottery study, and the 
CMO study. In the KIPP and CMO studies, the authors reported three-year impact estimates separately from one- 
year impact estimates. The three-year impact estimates for the charter lottery study were obtained by increasing the 
two-year impact estimates by 50 percent. 
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Figure 2. Charter school three-year impact estimates from various studies 



Kauffman Boston New York City KIPP charters Study of CREDO CMO study 

School charters charters low-income urban charter 

charter school study 
lotteries 

■ Mathematics ■ Reading 

Note: Figure 2 contains three-year effect size estimates for the average Boston charter middle school reported in 

Abdulkadiroglu et al. (2009), the average KIPP charter school analyzed by Tuttle et al. (2013), the average 
New York City charter school in grades 4 through 8 reported in Hoxby et al. (2009), the average charter 
school with a lottery admission process serving a large fraction of low-income students analyzed by 
Gleason etal. (2010), the average urban charter school in the 41 regions analyzed by the Centerfor 
Research on Education Outcomes (CREDO 2015), and the average school in the CMOs studied by 
Furgeson et al. (2012). 

CMO = charter school management organization; CREDO = Centerfor Research on Education Outcomes; KIPP = 
Knowledge Is Power Program. 


Relatively few studies of charter school effectiveness report impact estimates in science 
because there are fewer available data on science achievement than there are for mathematics 
and reading. New York City charter schools are estimated to have average annual impacts of 
0.23 standard deviations on science achievement in grades 5 through 8 (Hoxby et al. 2009). KIPP 
charter schools are estimated to have an average impact of 0.33 standard deviations in science 
three to four years after enrollment, implying an average one-year impact of approximately 0.094 
standard deviations. 16 The Kauffman School’s estimated science impact is substantially larger 
than both of these. 

Few of the charter schools included in the studies cited here were in their first two or three 
years of operation. Studies have shown that the performance of charter schools often improves 
after the first year of operation (Gill et al. 2007; Sass 2006). These results suggest that the 
Kauffman School’s estimated effects are especially noteworthy because the three-year impacts 
are based on students who enrolled during the first year the Kauffman School was open. 


16 


The KIPP one-year science impact estimate was obtained by dividing the three- to four-year impact estimate by 3.5. 
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V. THE IMPACT OF THE KAUFFMAN SCHOOL ON STUDENT ATTENDANCE 
AND SUSPENSIONS 


In this section, we present the impacts of the Kauffman School on student attendance and 
suspensions during the 2013-2014 school year. It is important to keep in mind that the analysis 
of suspensions cannot distinguish effects driven by differences in student behavior from effects 
driven by differences in the enforcement of school policies or reporting practices. If Kauffman 
students are more likely than students in other schools to be suspended, this could be due to poor 
behavior or because the Kauffman School issues suspensions for different kinds of behavior than 
other Kansas City schools. 

The set of comparison students used to analyze attendance and suspension outcomes is the 
same as the set used to analyze MAP achievement in Section IV. We also used the same set of 
baseline control variables, including 4th-grade attendance and suspension infonnation. We 
analyzed the attendance and suspension outcomes separately by cohort and by grade to highlight 
differences that arise over time and across grades (see Section VI of the appendix for impact 
estimates from previous school years). We present three sets of results for suspension outcomes. 
The state of Missouri collects suspension data separately for in-school suspensions and out-of- 
school suspensions. For the initial suspension analysis, we combined these data into one variable, 
indicating whether a student received either type of suspension. Our aim in combining these data 
was to create a variable that would be as comparable as possible across schools, because some 
schools have different standards for the kinds of disciplinary infractions that warrant in-school 
and out-of-school suspensions. To provide additional information on the source of the Kauffman 
School suspension impacts, we also present separate results using indicators for ever receiving an 
in-school or out-of-school suspension as outcome variables. (See Section I of the appendix for 
further details about the construction of the attendance and suspension variables.) 

The distribution of suspensions is skewed, with the majority of students receiving no 
suspensions. To simplify the analysis for the main suspension impact results, our outcome of 
interest is whether a student ever received a suspension during the year. 17 To assist with 
interpretation of the magnitude of the differences in suspension rates, we also present the average 
number of days suspended among Kauffman students and matched comparison students. We 
then discuss the relationship between the suspension rates at the Kauffman School and the large 
positive impacts the Kauffman School has on student achievement. 

Before presenting the impact estimates of the Kauffman School, we show some descriptive 
statistics about attendance and suspensions at the Kauffman School and other schools in Kansas 
City during the 2013-2014 school year. That descriptive information is displayed in Table 2. The 
attendance rate of Kauffman students was significantly higher than that of other Kansas City 
students in all grades. The Kauffman School also suspended students at a significantly higher 
rate during 2013-2014, compared with other schools in Kansas City. Almost half (48 percent) of 
Kauffman 5th-grade students received at least one suspension during the year, compared with 18 
percent of 5th-graders districtwide. Kauffman in-school and out-of-school suspension rates were 
28 and 22 percentage points higher than districtwide rates, respectively. Similarly, 48 percent of 


17 

Because the suspension outcome is a binary variable rather than a continuous one, we used a logit model in place 
of the linear regression to implement the analysis. 
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Kauffman 6th-grade students received at least one suspension during the year, compared with 23 
percent of 6th graders districtwide. Both in-school and out-of-school suspension were higher for 
Kauffman 6th graders. Approximately half (5 1 percent) of 7th-grade Kauffman students were 
suspended during 2013-2014, compared with 33 percent of 7th graders districtwide. In-school 
suspensions appear to drive the higher percentage of 7th graders receiving suspensions at the 
Kauffman School, as the percentage of students receiving an out-of-school suspension did not 
differ between Kauffman and district 7th graders. 

Table 2. Attendance and suspension rates for Kauffman and all Kansas City 


students during 2013-2014 


Kauffman students 

Kansas City students 

Difference 

5th-grade students 

Attendance rate (%) 

95.8 (4.4) 

94.8 (4.5) 

1.0“ 

Ever suspended (%) 

48.4 (50.1) 

17.5 (38.0) 

30.9“ 

Ever suspended (in-school) (%) 

35.7 (48.0) 

7.3 (26.0) 

28.4“ 

Ever suspended (out-of-school) (%) 

35.2 (47.9) 

13.1 (33.8) 

22.0“ 

Sample size 

182 

1,675 


6th-grade students 

Attendance rate (%) 

96.0 (3.3) 

94.6 (4.8) 

1 .4** 

Ever suspended (%) 

47.6 (50.2) 

22.9 (42.1) 

24.7“ 

Ever suspended (in-school) (%) 

31.0 (46.5) 

12.0(32.5) 

19.0“ 

Ever suspended (out-of-school) (%) 

34.5 (47.8) 

15.9(36.6) 

18.6“ 

Sample size 

84 

1,539 


7th-grade students 

Attendance rate (%) 

95.1 (5.1) 

93.0 (6.1) 

2.1“ 

Ever suspended (%) 

50.5 (50.3) 

32.7 (46.9) 

17.8“ 

Ever suspended (in-school) (%) 

40.0 (49.2) 

18.3(38.7) 

21.7“ 

Ever suspended (out-of-school) (%) 

29.5 (45.8) 

24.3 (42.9) 

5.2 

Sample size 

95 

1,505 



Note: Standard deviations are presented next to means in parentheses. Two asterisks (**) indicate results are 

significantly different at the 1 percent level. 


The descriptive statistics presented in Table 2 are based on the full sample of Kauffman and 
Kansas City students in 5th, 6th, and 7th grades during 2013-2014. In Table 3, we present the 
estimated impacts of the Kauffman School on attendance and suspensions during 2013-2014, 
based on the sample of matched comparison students. The impact estimates for other school 
years are displayed in Appendix Tables A. 5 and A. 6. 
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Table 3. Impact of Kauffman School on attendance and suspensions during 
2013-2014 


5th -grade 
students 

6th-grade 

students 

7th-grade 

students 

2013-2014 

average 

Attendance rate (%) 

0.57 (0.39) 

0.51 (0.46) 

1.16* (0.57) 

0.72** (0.27) 

Probability of being suspended (%) 

27.9** (3.6) 

24.9** (5.2) 

18.7** (5.5) 

24.7** (2.6) 

Probability of in-school suspension (%) 

29.1** (3.4) 

20.0** (5.1) 

21.1** (5.8) 

24.8** (2.6) 

Probability of out-of-school suspension (%) 

19.4** (4.0) 

20.9** (5.6) 

8.2 (5.1) 

16.6** (2.8) 

Sample size 

948 

585 

534 

2,067 


Note: The suspension results are marginal effects from logit models in which the outcome variable is an indicator 

for receiving a suspension during the year. Standard errors are displayed in parentheses beside each 
impact estimate. The sample size represents the total number of Kauffman students and matched 
comparison students entering each analysis. The fourth column represents a weighted average (by the 
number of Kauffman students) of the impacts across grade levels. One asterisk (*) indicates results are 
significantly different from zero at the 5 percent level. Two asterisks (**) indicate results are significantly 
different from zero at the 1 percent level. 


Impact on attendance 

The results show that during the 2013-2014 school year, the Kauffman School overall had a 
positive and significant impact on the attendance rate of its students. The magnitude of the 
coefficient indicates that, on average, Kauffman students had an attendance rate that was 0.72 
percentage points higher than that of other similar students in Kansas City. The estimated impact 
is positive in all grades, but is only significant for 7th-grade students where the magnitude of the 
impact is almost twice as large as it is in 5th or 6th grade. 

Impact on suspensions 

The estimated suspension impacts for the Kauffman School are also positive, indicating that 
Kauffman students are significantly more likely to be suspended than similar students in Kansas 
City. Overall, Kauffman students were 25 percentage points more likely to receive at least one 
in-school suspension and 17 percentage points more likely to receive at least one out-of-school 
suspension than were comparison students. 

The positive and significant impacts could be due to (1) stricter discipline policies at the 
Kauffman School, which might result in the issuing of suspensions for less-severe infractions 
than at other schools; (2) the longer school day and school year at the Kauffman School, which 
provides more opportunities for students to misbehave and for suspensions to be issued; or (3) a 
larger number of behavior problems from Kauffman students than comparison students. In 
Section VI, we discuss the behavior policies of the Kauffman School in detail, and show that the 
high suspension rates may be a result of the Kauffman School’s strict and consistently enforced 
discipline policy. 

Relationship between suspensions and student achievement 

In this section, we describe possible mechanisms through which the Kauffman School can 
have large positive impacts on achievement while suspending students at significantly higher 
rates than other schools. To help put the high suspension rates at the Kauffman School into 
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perspective, we present the average number of days suspended among students who receive at 
least one suspension in Table 4. The average number of days suspended among students ever 
suspended is not significantly different between Kauffman and matched comparison students in 
any of the three grades examined. The average number of days suspended among students 
receiving at least one suspension is between two and three for most grades for both Kauffman 
and comparison students. The results are similar when median days suspended is examined in 
place of average days suspended; the median days suspended ranges from one to two across 
grades for both groups. Overall, although more students receive suspensions at the Kauffman 
School, the average number of days missed due to suspensions among suspended students is not 
significantly higher among Kauffman students. 

The fact that the average number of days missed among Kauffman students who receive 
suspensions is approximately three helps to shed light on how the Kauffman School may be 
having large positive impacts on student achievement while suspending students at higher rates. 
The additional instructional time at the Kauffman School that results from the extended school 
day and year is much larger than three days. The Kauffman School estimates that students 
receive the equivalent of approximately five weeks per year of additional schooling relative to 
other public school students in Kansas City (see Section VI for details). Since the average 
number of days missed due to suspensions is small relative to the additional instructional time at 
the Kauffman School, it is possible that the high suspension rates are not substantially detracting 
from learning. It also may be the case that enforcement of the Kauffman School’s discipline 
policy through suspensions improves the future behavior of suspended students and minimizes 
classroom disruptions. 


Table 4. Average number of days suspended for students receiving 
suspensions during 2013-2014 



Kauffman students 

Kansas City students 

Difference 

5th-grade students 

Mean suspensions 

3.33 (3.20) 

2.38 (2.21) 

0.95 

Mean suspensions (in-school) 

2.26 (1.74) 

2.52 (2.17) 

0.70 

Mean suspensions (out-of-school) 

2.26 (1.86) 

1.80 (1.26) 

0.25 

Sample size 

76 

137 


6th-grade students 

Mean suspensions 

2.64 (1.82) 

2.20 (1.94) 

0.44 

Mean suspensions (in-school) 

1.92 (1.29) 

2.02 (1.59) 

-0.10 

Mean suspensions (out-of-school) 

1.84 (1.15) 

1.66 (1.16) 

0.17 

Sample size 

36 

119 


7th-grade students 

Mean suspensions 

2.82 (3.18) 

2.92 (2.60) 

-0.10 

Mean suspensions (in-school) 

1.93 (1.42) 

2.56 (2.13) 

-0.62 

Mean suspensions (out-of-school) 

2.18 (2.20) 

2.01 (1.53) 

0.17 

Sample size 

47 

141 



Note: Standard deviations are presented next to means in parentheses. The sample size represents the total 

number of Kauffman students and matched comparison students with at least one suspension (in-school or 
out-of-school). No differences are statistically significant at the 5 percent level. 
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VI. SCHOOL HALLMARKS DESCRIPTION AND PERCEPTIONS OF 
IMPLEMENTATION 


In this section, we discuss the Kauffman School’s hallmarks to help address the question of 
what factors are contributing to the high level of student achievement growth at the Kauffman 
School. We describe the Kauffman School’s hallmarks and its key stakeholders’ (school 
administrators, teachers, students, and parents) perceptions of how the Kauffman School’s 
hallmarks are being implemented. 

The descriptive information on the Kauffman School’s hallmarks and their implementation 
is derived from various sources of data collected while conducting annual site visits three times 
per school year. The types of data collected include: (1) School and Foundation documents and 
interviews with school administrators; (2) focus group discussions with teachers, students, and 
instructional coaches; (3) classroom observations; (4) surveys of teachers, parents, and students; 
and (5) interviews with parents. More detail about these data collection efforts can be found in 
Section II of the appendix. 

High attendance and behavioral expectations 

Recognizing that students must be in school in order to meet its academic growth targets, the 
Kauffman School set 95 percent as the daily attendance goal. Upon enrollment in the Kauffman 
School, students and families are informed that students should have fewer than 10 absences 
each year and those with 10 or more absences, excused and unexcused, may be retained in their 
current grade. The Kauffman School advises students and parents that colleges review students’ 
attendance records as part of the application process, which is another reason to minimize 
absences. 

In addition, the Kauffman School implemented a system of behavioral expectations and 
school rules designed to minimize distractions and maximize instructional time. According to the 
Kauffman School’s Handbook (Ewing Marion Kauffman School 2014), the Kauffman School’s 
discipline system utilizes merits to reward student behavior (for example, displaying leadership 
or good citizenship) and demerits for non-compliance with School policies and procedures (for 
example, not following directions, or violating the unifonn code). Students may be required to 
serve detention during school hours as a result of earning three demerits in one day (Ewing 
Marion Kauffman School 2013). Students are expected to demonstrate active behavioral and 
cognitive engagement following the SLANT method (Sit up, Listen, Ask and answer questions, 
Nod your head, and Track the speaker) During class, students are expected to speak in a voice 
that is confident and loud enough for everyone to hear — what the Kauffman School refers to as 
their “college voice.” 

During the study’s classroom observations, teachers were observed actively tracking and 
documenting students’ behavior, providing students with behavioral cues related to SLANT (for 
example, “Let’s all track Johnnie.”), and offering silent reminders about appropriate behavior 
(for example, standing next to a student who is off-task). From the perspective of parents, the 
Kauffman School’s behavior expectations and rules are beneficial to students. When asked about 
SLANT, one parent remarked that SLANT works because it helps her child engage and 
“...absorb it all.” 
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The Kauffman School also makes considerable efforts toward heightening awareness of the 
school rules. The Kauffman School has a week-long orientation for new students at the 
beginning of the school year and one of the primary goals of that orientation is to introduce the 
behavioral expectations and rules the Kauffman School has for students. Key stakeholders at the 
Kauffman School report that students have a high awareness of the school rules. The vast 
majority of students agreed that they know the school rules (87 percent) and that students know 
what the punishment will be if they break a school rule (84 percent). Similarly, the vast majority 
of teachers report that students know the rules (93 percent) and know what the consequences will 
be if they break a rule (85 percent). However, student and teacher survey responses indicate 
students’ adherence to the rules is not as strong. For example, less than half of students (47 
percent) agreed that most students follow the rules at school and about one-third of teachers (37 
percent) agreed that student discipline is a problem in their classroom. These reports on student 
misbehavior may also be related to the Kauffman School’s high suspension rates that were 
previously discussed. When administrators and teachers sense persistent school-level behavioral 
issues, the school staff work together to ensure that routines and procedures are consistently 
enforced. 

As with other aspects of the Kauffman School, the discipline system, including the 
suspension policy, was informed by other successful charter schools’ policies. The Kauffman 
School sought to establish clear guidelines for suspensions in which everyone knows the 
consequences for specific behaviors. School administrators meet with parents and families when 
students engage in behaviors that have been established as warranting suspension to discuss the 
tenns of the suspension, but students cannot avoid being suspended. From the Kauffman 
School’s perspective, consistent application of its suspension policy helps to strengthen its 
culture. 

Ambitious academic goals 

According to the Kauffman School’s charter application, the Kauffman Foundation expected 
that incoming students would be one to three years behind grade level. The Foundation estimated 
that students, on average, would need to achieve 1 .25 years of academic growth during each year 
of enrollment to be at grade level by the end of 8th grade and prepared for high school. The 
Foundation therefore set 1.25 years of academic growth as an explicit charter goal for the 
Kauffman School to achieve. 

In focus groups, teachers frequently cited the Kauffman School’s high academic goals as 
influencing their decision to work at the Kauffman School. Teachers feel that students have 
internalized the high academic expectations. The majority of Kauffman students (67 percent) 
agreed that their classes are challenging and nearly all parents (98 percent) agreed that their child 
was challenged at school. The academic goals were also perceived by teachers and parents as 
being realistic and appropriate for Kauffman School students. Nearly all teachers (93 percent) 
agreed that most of their students can learn everything they want them to leam during the year. 
Nearly all parents (94 percent) agreed that the standards set by the school are realistic. 


23 



WORKING PAPER 43 


MATHEMATICA POLICY RESEARCH 


Extended school day and year 

The Kauffman School’s design team anticipated that ensuring 1.25 years of growth each 
school year would require additional instructional time. During the Kauffman School’s first year, 
students received 37.8 hours of instruction per week (Richardson 2009). In the Kauffman 
School’s second year, the instructional time was shortened to 36.5 hours per week in response to 
feedback from students, parents, teachers, and staff that the longer school day was too difficult 
for everyone to manage. In the third year of operation, the instructional time was slightly reduced 
from the second year to 36.2 hours per week. According to the Kauffman School calendar, there 
were also seven additional days of instruction during the school year, which when combined 
with the longer school days equates to approximately five additional weeks of school per year 
compared to traditional public schools in Kansas City. 

With the longer school day, the Kauffman School provides students with opportunities for 
additional learning and relationship-building with their teachers and peers. For example, students 
receive additional instructional support during the Kauffman School’s “Focus” class period. 
During Focus, struggling students receive additional instruction and practice in any subjects in 
which they need help, and high-performing students can receive advanced instruction. The Focus 
class is approximately 35 minutes long and is offered during the last period of every school day. 

During focus groups, some students expressed appreciation for the longer school day stating, 
“[My] favorite thing about the Kauffman School is [that] we have more time to learn.” 

According to survey data, students and teachers generally perceived the longer school day and 
year as being beneficial. Over half of students (55 percent) agreed that they learn more because 
they have a longer school day and over two-thirds of teachers (68 percent) agreed that they 
believed the longer school day and year benefits their students. 

Increased mathematics and reading instructional time 

The school design team determined that its students would need more instructional time 
specifically in mathematics and reading to catch up academically. Each day in Year 1, Kauffman 
students attended a double period of mathematics (104 minutes), a nonfiction reading class (50 
minutes), a writing class (50 minutes), and an extended period of literature study (80 minutes). 
During Years 2 and 3, students continued to have a double period of mathematics (100 minutes) 
and three periods (150 minutes) of instruction related to ELA and reading. 

Intensive data-driven decision making 

The Kauffman School has a strong emphasis on data and results, and employs a large 
assessment portfolio that it uses to make data-driven decisions. Teachers receive training to 
understand and use the various data sources to monitor students’ learning and to adjust 
curriculum and instruction as necessary. Teachers reported regularly reviewing students’ 
assessment data to determine “. . .what skills need to be re-taught and how to re-teach them.” 

One mathematics teacher described re-teaching a unit on fractions because students’ ANet scores 
indicated that they “. . .were missing crucial conceptual knowledge about fractions.” When asked 
about their professional growth at the Kauffman School, teachers frequently mention data-driven 
decision-making, “I feel I have grown a lot in looking at, analyzing, and making very aligned 
decisions about instruction based on what our kids need.” On the teacher surveys, more than half 
of teachers (56 percent) rated themselves as advanced proficient in their ability to use assessment 
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data to purposefully plan their instruction. The remaining teachers (44 percent) rated themselves 
as beginning proficient in their ability to use assessment data, with no teachers rating themselves 
as a novice. From the perspective of teachers, data-driven decisions contribute to their students’ 
success in the classroom. 

In addition to assessment data, the Kauffman School also tracks and monitors students’ 
behavioral data (for example, merits, demerits, detentions). Teachers described keeping a 
“. . .frequent pulse on [behavior],” which helps them readily address behavior problems as they 
emerge. As part of this effort, professional development sessions often focus on classroom 
management and student behavior. 

Extensive teacher professional development 

The Kauffman School places a significant emphasis on teachers’ professional development. 
Kauffman School teachers are frequently observed by and receive feedback from administrators, 
have weekly meetings with instructional coaches, and participate in weekly small- and large- 
group professional development sessions. All School staff, including teachers, attend a two-week 
professional development session every summer. The Kauffman School also promotes classroom 
observations through the physical configuration of the building: the classrooms were designed to 
have large interior windows that allow any adult in the hallway to see what is happening in any 
classroom. 

According to teacher survey data, all teachers reported receiving professional development 
in the previous month, and nearly all experienced different types of professional development 
from multiple sources. Nearly all teachers (96 percent) reported being mentored or coached by 
an administrator and over three-quarters (8 1 percent) reported being mentored or coached by 
another teacher. Teachers also reported having someone observe their classroom teaching five 
times in the previous month (median=three times by a peer or colleague; median=two times by a 
mentor or supervisor). Teachers reported meeting formally or informally nine times in the past 
month with peers or colleagues and six times with mentors or supervisors. Teachers stated they 
received written feedback on their perfonnance two times in the previous month from peers or 
colleagues and two times from mentors or supervisors. The rate at which teachers are observed 
and receive feedback is high even when compared to many other charter schools. Only 17 
percent of the charter school principals surveyed in Furgeson et al.’s (2012) study of CMOs 
reported that administrators observe their teachers eight or more times in a year, while 61 percent 
of teachers were observed between four and eight times and 22 percent were observed between 
two and four times. 

Incoming and returning teachers frequently cite the Kauffman School’s professional 
development program as one of the main things that attracted them to the school. One teacher 
shared, “for me it was the opportunity to grow as a teacher. . .getting the feedback to become 
better, that’s what really attracted me to come here.” In general, teachers are satisfied with the 
professional development they receive and frequently comment on their professional growth. In 
the words of one teacher, “last year I improved more as a teacher than I could have imagined I 
would.” 
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Well-established cultural norms 

The Kauffman School takes an intentional approach to establishing a culture that consists of 
shared values, expectations, and norms. School leaders continuously communicate the values, 
expectations, and norms to all school staff, students, and families. The school design team laid 
out its expectations for the Kauffman School’s culture and the importance of a strong school 
culture in their early planning documents: “Research, school visits, and advice from charter 
school practitioners and experts all emphasize the vital importance of a strong culture as essential 
to school success.” The team highlighted some of their expectations for the Kauffman School’s 
culture, including, but not limited to: (1) a clearly defined and consistently implemented 
discipline system; (2) visible and frequently communicated “school vocabulary”; (3) school 
uniforms or other special clothing; (4) required family meeting for all incoming students before 
attending the Kauffman School; and (5) disciplinary merit/demerit systems. The hallmarks 
discussed to this point reflect the Kauffman School’s implementation of these expectations. 

For School leaders, consistent implementation of these elements is key to the Kauffman 
School’s ability to maintain a strong and readily visible culture. For example, students’ uniforms 
are inspected daily to ensure strict adherence. The Kauffman School also established common 
classroom procedures (for example, every teacher writes the daily learning target on the board) 
that are implemented across classrooms to ensure that students are having a consistent classroom 
experience. In addition, the Kauffman School engages in “cultural resets” with staff and students 
that aim to refocus everyone on the Kauffman School culture. For example, if it seems that 
teachers are not implementing the merits and demerits consistently or students are not adhering 
to school policies and norms, school leaders hold cultural reset meetings with staff and students. 
The teacher survey data indicate that there is a strong sense of a collective unity among teachers, 
with all teachers agreeing that there is a highly cooperative effort among the members of the 
Kauffman School staff and that the staff share the same beliefs and values about the central 
mission of the school. For parents, the Kauffman School’s consistency is evident and parents 
note that “the rules stay the same.” 

VII. CONCLUSION 


The Kauffman School has positive, statistically significant, and educationally meaningful 
impacts on student achievement growth in mathematics, reading, and science. The estimated 
impacts for students three years after enrollment are 0.57 standard deviations in mathematics and 
0.41 standard deviations in reading, equivalent to approximately 1.35 additional years of learning 
growth in mathematics and 1.29 additional years of learning growth in reading. In science, the 
one-year impact estimates are similarly high, equal to approximately 1 .08 additional years of 
learning growth. Students who enroll in the Kauffman School were substantially below the state 
average in terms of mathematics and reading achievement prior to enrolling but achieve above 
average on state assessments three years later. 

Average attendance rates at the Kauffman School are above 95 percent in all grades. 
Suspension rates are consistently higher for Kauffman students relative to comparison students. 
Although the likelihood of receiving at least one suspension was approximately 25 percentage 
points higher among Kauffman students, the average number of days suspended among those 
receiving suspensions was similar between Kauffman and comparison students. The high 
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suspension rates at the Kauffman School are likely driven by high behavioral expectations and 
strict enforcement of the discipline policies associated with them. Although high suspension 
rates are often associated with lower achievement growth, the cost of missed instruction time due 
to suspensions may be outweighed by the benefit of maintaining strong discipline and removing 
disruptive students from the classroom. Moreover, the average duration of suspensions was small 
compared to the increased instruction time experienced by Kauffman students due to the 
Kauffman School’s extended school day and year. 

We explored seven hallmarks of the Kauffman School’s practices to shed light on the 
mechanisms that might be contributing to these impacts. Although we cannot definitively 
detennine which practices are providing the largest contribution to the Kauffman School’s 
positive impacts on student achievement, placing the Kauffman School’s policies in context with 
practices shown to be successful in the research literature can serve as a guide. The Kauffman 
School provides frequent intensive professional development for teachers, which prior research 
has found to be related to high achievement growth for students. Part of the teacher professional 
development at the Kauffman School involves training in the use of student assessment data to 
improve instruction, which has also been shown to be associated with high achievement growth 
for students. The positive impacts of the Kauffman School may also be related to the fact that the 
Kauffman School has an extended school day and year, which allows for additional instructional 
time in mathematics and reading. Increased instructional time combined with ambitious 
academic goals serve to encourage strong academic achievement growth at the Kauffman 
School. 

The final hallmark of the Kauffman School explored is that the Kauffman School has well- 
established cultural norms. School leaders feel that maintaining a strong culture creates an 
environment in which teaching and learning can flourish because there is a shared understanding 
of the Kauffman School’s values, expectations, and norms among all stakeholders. The 
Kauffman School is deliberate in its implementation and oversight of its culture to ensure that all 
aspects of the culture are consistently implemented, distractions are minimized, and time for 
instruction and learning is maximized. 
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I. DATA PREPARATION DETAILS FOR IMPACT ANALYSIS 


The Missouri Department of Elementary and Secondary Education provided data on state 
assessment results and student characteristics for all students enrolled in Missouri public schools 
during our analysis period. Mathematica Policy Research requested student data for all 3rd 
graders in 2010-2013, all 4th graders in 2011-2013, all 5th graders in 2012-2013, all 6th graders 
in 2013-2014, and all 7th graders in 2014. The state assessment data contained Missouri 
Assessment Program (MAP) scaled scores, proficiency levels, and test accommodation 
infonnation for each student test by year, grade level, and content area. The student 
characteristics data contained demographic, free or reduced-price lunch status, limited English 
proficiency, disability, attendance, and disciplinary infonnation for each student by year and 
school of enrollment, as well as school-level characteristics such as charter school classification 
and school location. 

To link the state assessment and student characteristics data, we reduced both to the unique 
student level. In the state assessment data, we removed records in which students had more than 
one unique subject-specific MAP scaled score reported in a given year. In the characteristics 
data, we first removed all records with zero or missing reported attendance and then summed 
attendance and disciplinary variables across each student’s school-specific records to calculate 
student-year totals. We then reduced the data to the student level such that all year-specific 
attendance/disciplinary information was preserved in separate variables, and demographic, free 
or reduced-price lunch status, limited English proficiency, and disability infonnation was taken 
from the student’s 4th-grade record if available, 3rd-grade record if the 4th-grade record was 
unavailable, and 5th-grade record if both 4th- and 3rd-grade records were not available. Any 
students not found in both the characteristics and state assessment data were dropped from the 
analysis. 

We created several new variables to facilitate the analyses. We transformed student MAP 
scaled scores into z-scores based on statewide year-, grade-, and subject-specific means and 
standard deviations. We also used enrollment and absence infonnation to create an attendance 
rate measure that we bottom-coded at the year-specific first percentile to remove extreme 
outliers. We used disciplinary information to create yearly indicators of whether students 
received a suspension that year. 18 We then collapsed subject-specific 3rd- and 4th-grade MAP z- 
scores into grade-specific variables by taking each student’s most recent score (by year) within 
grade level for those students repeating grades. We created a single binary test accommodation 
indicator to represent having test accommodations on any 3rd- or 4th-grade MAP test. 

II. DATA FOR ANALYSIS OF SCHOOL HALLMARKS 


Throughout the Kauffman School’s first three years of operation, Mathematica collected a 
variety of data which are used to describe the Kauffman School’s hallmarks and perceptions of 
their implementation. Data reported in this study are from those collected during three-day site 
visits to the Kauffman School. During the site visits, we conducted in-depth interviews with 


18 

All analyses use 4th-grade attendance and suspensions as control variables. If 4th-grade information on these 
variables was missing, then 3rd-grade values were used instead. 
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School administrators and Foundation staff; conducted focus groups with teachers, instructional 
coaches, and students; conducted one-on-one interviews with parents; and administered surveys 
to teachers, students, and parents. The Kauffman Foundation also shared documentation related 
to the Kauffman School’s start-up activities, such as its charter application, memos to the 
Foundation’s Board of Trustees related to School planning and design, and teacher recruitment 
materials. 

Procedures 

At the time of application to the Kauffman School, parents agreed to pennit student records 
to be used for research purposes. A second parental consent form, specific to the other data 
collection activities, was distributed during the first week of school. This consent fonn described 
the evaluation and data collection activities in greater detail. Ninety-eight percent of parents 
consented to participate and we also obtained each student’s assent before he or she completed 
the student surveys. Teachers, School administrators, and Foundation staff were also asked to 
sign a consent form agreeing to participate in the study. The teachers’ consent form included the 
provision that their responses to survey questions and during focus group discussions would 
remain anonymous. 

Interviews with School administrative staff 

In-depth interviews, ranging from 1 5 to 45 minutes with various administrative staff at the 
Kauffman School covered topics related to: (1) general observations and impressions of the 
Kauffman School and its relationship to the Foundation, (2) recruitment and retention of staff 
and work-life balance, (3) School culture and leadership structure, (4) curriculum, instruction, 
and assessment, (5) student discipline, and (6) plans for the future. 

Surveys and focus groups with teachers 

We conducted two types of data collection with Kauffman School teachers: surveys and 
focus groups. Self-administered teacher surveys were completed in the spring by 27 out of 3 1 
teachers (87 percent) at the Kauffman School. Teachers were asked questions about their 
students, curriculum and instruction, their experiences teaching at the Kauffman School, parent 
involvement, and overall satisfaction with the Kauffman School. These surveys were emailed to 
teachers before each site visit and teachers were asked to return the surveys by the end of the site 
visit. In addition to the survey, teachers were randomly selected to participate in focus group 
discussions during the site visit. These discussions addressed a variety of topics including: staff 
recruitment and retention, curriculum, instruction and assessment, School culture and structures, 
teacher-student relationships, student and teacher growth and development, overall reflections 
and observations of the Kauffman School, and plans for coming school year. The focus groups 
were scheduled at a time that was convenient for teachers. At the conclusion of each focus group, 
teachers received a $50 gift card for their participation. 

Surveys of parents 

Parent surveys were administered by Kauffman Foundation staff on report card night, which 
occurred in the weeks before the site visit to the Kauffman School. The 15-minute survey 
included questions related to school choice, their child’s study habits and behaviors, their plans 
for their child’s future, and their overall satisfaction with the Kauffman School. The survey was 
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available in both English and Spanish. Parents who did not complete the survey at report card 
night were sent a letter asking them to complete the survey and send it back to Mathematica in a 
pre-paid envelope. A small incentive was offered to help increase the parent response rates. Only 
135 parents out of 368 responded to the survey (38 percent). 

Surveys and focus groups with students 

We conducted two types of data collection activities with Kauffman School students: 
surveys and focus groups. The surveys asked students questions about their learning, motivation 
levels, activities and homework, goals for the future, behavior outside of school, school climate, 
and satisfaction with the Kauffman School. Surveys were administered by School staff during 
the school day at the time of the site visit. The response rate for the student survey was 60 
percent (221 out of 368 students). In addition to the student surveys, we held focus groups with 
students during the site visit. One focus group was conducted separately for each grade level, and 
two male and two female students were randomly selected from each grade to participate in the 
focus group discussions. Focus groups were conducted during the students’ lunch time and lasted 
between 15 and 20 minutes. The discussions addressed students’ general reflection on their time 
at Kauffman, specifically their likes and dislikes about the Kauffman School, their satisfaction 
with the Kauffman School, comparisons to prior schools, and their plans for staying at the 
Kauffman School in the coming years. 

Classroom observations 

During the spring site visit, Mathematica staff observed a random selection of 14 teachers. 
Trained Mathematica observers identified and coded activities and behaviors in the classroom, 
paying particular attention to student engagement, evidence of School values, peer respect, 
teacher and student respect, discipline, classroom instruction, and teacher feedback. 

III. SAMPLE SELECTION FOR IMPACT ANALYSIS 


The Kauffman School group is composed of students enrolled in the Kauffman School in 
5th grade in 2011-2012, 2012-2013, or 2013-2014 for at least part of the school year. The 
comparison group is composed of students enrolled during at least part of the school year in a 
district or charter school in Kansas City who were not included in the Kauffman School group. 
We developed a list of charter schools using information on school location and background 
research on charter schools identified as being located in Kansas City (Missouri Department of 
Elementary and Secondary Education 2014) and enrolling 5th-grade students in our analysis 
years. 

In addition to these restrictions, we excluded any Kauffman students who were missing all 
outcome (5th, 6th, or 7th grade) grade MAP test scores or all 3rd- and 4th-grade MAP test 
scores. 19 We also dropped any comparison students missing all 3rd- and 4th-grade MAP test 
scores or any outcome scores from the final analysis sample. For analyses based on data from 
2013-2014, 7 Kauffman students (7 percent) were dropped from the cohort I analysis sample, 18 
Kauffman students (18 percent) were dropped from the cohort II analysis sample, and 16 


19 Students who transfer to different school districts in Missouri will generally remain in our sample, but students 
who leave the state will be excluded due to missing outcome test scores. 


A.4 
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Kauffman students (9 percent) were dropped from the cohort III analysis sample as a result of 
these restrictions. With respect to the full comparison group, 21 percent of otherwise eligible 
students were dropped from the cohort I analysis sample, 16 percent of otherwise eligible 
students were dropped from the cohort II analysis sample, and 1 1 percent of otherwise eligible 
students were dropped from the cohort III analysis sample. See Johnson et al. (2014) for details 
about the percentage of students dropped from the analyses based on data from 2012-2013. 
Finally, we excluded any students who were enrolled for part of any school year at the Kauffman 
School from the comparison group (and included them in the Kauffman School group). The 
numbers of students included in the Kauffman School and comparison groups for each grade and 
cohort in our analysis are provided in Table A. 1 , 20 


Table A.1. Number of students in each comparison group in 2013-2014 



Cohort 1 7th 

Cohort II 6th 

Cohort III 5th 

Final study group 

graders 

graders 

graders 

Kauffman students 

95 

84 

182 

All Kansas City public schools comparison 
group 

1,505 

1,539 

1,675 


We display the baseline average characteristics of all students included in the Kauffman 
School and comparison group, separately by cohort, in Table A. 2. This table shows that 
Kauffman students tend to differ significantly from students enrolled in Kansas City public 
schools on several key baseline measures. Kauffman students had significantly higher average 
4th-grade MAP test scores than students in other Kansas City schools, though both groups had 
test scores that were below the statewide average. 21 Kauffman students are also more likely to be 
black and less likely to be Hispanic, are less likely to receive baseline test accommodations, and 
had higher 4th-grade attendance rates relative to students enrolled in Kansas City schools. 22 
Kauffman students were generally similar to other Kansas City students with respect to free or 
reduced-price lunch status and disability status, though there were some significant differences 
across cohorts. 


20 

" See Johnson et al. (2014) for the corresponding numbers based on data from 2012-2013. The sample sizes for 
cohort I and cohort II students in Table A.l are smaller due to missing outcome data for some students in 2013- 
2014. 

2 1 

~ The statewide average (pooled across the years 2011-2012 through 2013-2014) 4th-grade MAP scaled score was 
648 in mathematics and 661 in reading. The statewide standard deviation of 4th-grade MAP scores was 34 in 
mathematics and 38 in reading. These numbers are calculated by averaging the year-specific means and standard 
deviations from 2011-2012 through 2013-2014. 

22 

~ Examples of test accommodations include extended test time, individual testing, and oral reading of test 
questions. 
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Table A.2. Baseline 4th-grade average characteristics of Kauffman School 
students and other Kansas City Public School students 


Kauffman School 

Other public schools in 
Kansas City 

Cohort 1 7th graders 

4th-grade mathematics scaled score 

636 

628* 

4th-grade reading scaled score 

651 

638** 

Free or reduced-price lunch 

0.86 

0.89 

Black 

0.79 

0.63** 

Hispanic 

0.14 

0.25** 

Male 

0.49 

0.49 

Disabled 

0.07 

0.09 

Any prior test accommodation 

0.08 

0.21** 

4th-grade attendance rate 

0.95 

0.94** 

4th grade ever suspended 

0.20 

0.18 

Sample size 

95 

1,505 

Cohort II 6th graders 

4th-grade mathematics scaled score 

643 

633** 

4th-grade reading scaled score 

651 

640** 

Free or reduced-price lunch 

0.80 

0.89* 

Black 

0.77 

0.61** 

Hispanic 

0.08 

0.24** 

Male 

0.51 

0.48 

Disabled 

0.12 

0.08 

Any prior test accommodation 

0.14 

0.24* 

4th-grade attendance rate 

0.96 

0.95** 

4th grade ever suspended 

0.12 

0.19* 

Sample size 

84 

1,539 

Cohort III 5th graders 

4th-grade mathematics scaled score 

636 

632 

4th-grade reading scaled score 

653 

639** 

Free or reduced-price lunch 

0.89 

0.90 

Black 

0.80 

0.58** 

Hispanic 

0.12 

0.27** 

Male 

0.44 

0.51 

Disabled 

0.05 

0.10* 

Any prior test accommodation 

0.12 

0.29** 

4th-grade attendance rate 

0.96 

0.95** 

4th grade ever suspended 

0.14 

0.15 

Sample size 

182 

1,675 


Note: One asterisk (*) indicates significantly different from Kauffman students at the 5 percent level. Two 

asterisks (**) indicate significantly different from Kauffman students at the 1 percent level. 
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IV. MULTIPLE IMPUTATION METHODOLOGY 


We calculated impact estimates using a multiple imputation procedure with M = 10 imputed 
data sets. We imputed missing baseline outcome variable values separately by treatment or 
comparison status using a chained linear equations model that included all outcome variables and 
all student characteristic variables included in the final impact regressions. 23 

Students were excluded from the imputation model if they had missing data for all 3rd- or 
4th-grade MAP test scores or missing data for all outcome (5th-, 6th-, or 7th-grade) MAP test 
scores. Missing values were imputed before propensity-score matching and regression analyses 
in each multiple imputation data set. 


After collecting coefficient and standard error estimates from each of the 10 imputed data 
sets, we computed multiple imputation coefficients and standard errors using Rubin’s 
combination method (Rubin 1987). The multiple imputation beta (J3m) coefficient is the average 
of the beta coefficient values in each imputed data set (fi m )\ the multiple imputation standard 
error is the square root of the within-imputation coefficient variance (Varw) plus the between- 
imputation coefficient variance ( Vara) inflated by a finite imputation correction multiplier: 


CD se m 
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1 + - 


M 


Var„ 



+ 

M + — 

M 1 


l M) 
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V. PROPENSITY-SCORE MATCHING METHODOLOGY 


We estimated a propensity score for each eligible treatment and comparison student in each 
multiple imputation data set using a stepwise logistic regression model. We used an entry 
criterion of (p < .20) to detennine whether each variable would enter the final logistic regression 
model. (See Table A. 3 for a list of the variables.) 

Table A.3. List of potential covariates used for propensity-score matching 

4th-grade mathematics and reading MAP z-scores 

Second- and third-order polynomials of 4th-grade mathematics and reading MAP z-scores 
3rd-grade mathematics and reading MAP z-scores 
4th-grade attendance rate and ever-suspended variables 

Gender, race, individualized education plan, English language learner, free or reduced-price lunch, any baseline test 
accommodation 

Interactions of 4th-grade mathematics and reading MAP z-scores with gender, race, individualized education plan, 

English language learner, free or reduced-price lunch, any baseline test accommodation 

Interactions of race with gender and free or reduced-price lunch 

Indicators for imputed 3rd- and 4th-grade mathematics and reading MAP z-score variables 


7T . 

~ In previous years, to maximize sample size we included all students in the state in the imputation of missing data 
for comparison group students. This year, to account for possible differences in the relationship among variables for 
Kansas City students relative to other districts in the state, we restricted the imputation sample for comparison 
students to include only other students in Kansas City who were part of the comparison group. The results were very 
similar with and without this restriction imposed, so we did not re-estimate the results from previous years. 
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Indicator for imputed 4th-grade attendance rate or ever-suspended variables 


After generating propensity scores for each Kauffman student and eligible comparison 
student, we selected a matched comparison group by finding comparison students with 
propensity score values within a given threshold, or radius, from each Kauffman student’s p- 
score. Comparison students were sampled with replacement, meaning that each comparison 
student could be matched to multiple Kauffman students. To limit the number of possible 
comparison students, we specified a minimum matching radius and maximum number of 
potential matched neighbors. The matching radius was larger for cohort II and cohort III 
Kauffman students relative to cohort I students, because these students differed more relative to 
comparison students on baseline characteristics compared with cohort I students (see Table A. 2). 
If there were no comparison students within the matching radius for a given treatment student, he 
or she was excluded from the matched comparison impact analyses. Because each comparison 
student could be matched to multiple treatment students, we used a weighting scheme in which 
each treatment student had a weight of one and each comparison student had a weight 
representing the number of treatment students matched to him or her. Table A. 4 gives summary 
matching infonnation for each comparison group. 24 


Table A.4. Matching information summary 


Match statistics 

Cohort 1 7th graders 

Minimum matching radius 

0.0003 

Maximum number of matches 

20 

Number of Kauffman students 

95 

Mean number of Kauffman students matched 

91 

Mean number of comparison students 

444 

Mean matches per Kauffman student 

6.2 

Cohort II 6th graders 

Minimum matching radius 

0.0005 

Maximum number of matches 

20 

Number of Kauffman students 

84 

Mean number of Kauffman students matched 

73 

Mean number of comparison students 

511 

Mean matches per Kauffman student 

8.7 

Cohort III 5th graders 

Minimum matching radius 

0.0005 

Maximum number of matches 

20 

Number of Kauffman students 

182 

Mean number of Kauffman students matched 

159 

Mean number of comparison students 

789 

Mean matches per Kauffman student 

6.9 


24 


See Johnson et at. (2014) for the matching information for other grade/cohort combinations. 
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In Table A. 5, we present summary statistics to show how well Kauffman students were 
matched to comparison students on baseline characteristics. On average, comparison students 
from each matched group were not significantly different from Kauffman students on any 
baseline characteristics used in the analysis. Note that the sample sizes in Table A. 5 are smaller 
for both Kauffman and comparison students relative to those in Table A. 2. This is because some 
Kauffman students differed enough from all comparison students such that no good match for 
these students could be found. The matched comparison analysis excludes these Kauffman 
students. (See Johnson et al. [2016] for a sensitivity analysis that includes these students.) 

Table A.5. Baseline 4th-grade average characteristics of matched 
comparison samples 


Kauffman School 

Other public schools in Kansas 

City 

Cohort 1 7th graders 

4th-grade mathematics scaled score 

635 (32) 

636 (33) 

4th-grade reading scaled score 

650 (34) 

651 (33) 

Free or reduced-price lunch 

0.86 (0.34) 

0.85 (0.36) 

Black 

0.78 (0.41) 

0.77 (0.42) 

Hispanic 

0.14 (0.35) 

0.14 (0.34) 

Male 

0.49 (0.50) 

0.44 (0.50) 

Disabled 

0.07 (0.26) 

0.07 (0.26) 

Any prior test accommodation 

0.09 (0.28) 

0.09 (0.29) 

4th-grade attendance rate 

0.95 (0.04) 

0.95 (0.04) 

4th grade ever suspended 

0.20 (0.40) 

0.16 (0.37) 

Sample size 

91 

444 

Cohort II 6th graders 

4th-grade mathematics scaled score 

642 (28) 

643 (28) 

4th-grade reading scaled score 

652 (31) 

650 (32) 

Free or reduced-price lunch 

0.87 (0.34) 

0.87 (0.33) 

Black 

0.77 0.42) 

0.75 (0.43) 

Hispanic 

0.10 (0.30) 

0.09 (0.29) 

Male 

0.53 (0.50) 

0.45 (0.50) 

Disabled 

0.11 (0.31) 

0.09 (0.28) 

Any prior test accommodation 

0.12 (0.33) 

0.15 (0.36) 

4th-grade attendance rate 

0.96 (0.03) 

0.96 (0.04) 

4th grade ever suspended 

0.14 (0.35) 

0.13 (0.33) 

Sample size 

73 

511 
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Other public schools in Kansas 



Kauffman School 

City 

Cohort III 5th graders 

4th-grade mathematics scaled score 

634 (31) 

635 (31) 

4th-grade reading scaled score 

647 (32) 

649 (33) 

Free or reduced-price lunch 

0.89 (0.31) 

0.90 (0.30) 

Black 

0.77 (0.42) 

0.77 (0.42) 

Hispanic 

0.13 (0.34) 

0.15 (0.35) 

Male 

0.44 0.50) 

0.47(0.50) 

Disabled 

0.06 (0.24) 

0.05 (0.21) 

Any prior test accommodation 

0.13 (0.34) 

0.14 (0,35) 

4th-grade attendance rate 

0.95 (0.04) 

0.95 (0.04) 

4th grade ever suspended 

0.15 (0.36) 

0.14 (0.34) 


Sample size 159 789 


Note: Standard deviations are displayed in parentheses next to the averages in this table. No differences 

between averages for Kauffman School students and comparison group students are significantly different 
from zero. 


IV. ADDITIONAL ATTENDANCE AND SUSPENSION IMPACT ESTIMATES 


Table A. 6 displays the attendance and suspension impacts for the first three cohorts of 5 th- 
grade students. Estimated impacts on attendance were not significantly different from zero for 
any of the three cohorts. Fifth-grade Kauffman students were suspended at a significantly higher 
rate than comparison students in 2011-2012 and 2013-2014 but not during 2012-2013. Table 
A. 7 displays a similar comparison of the first two cohorts of 6th-grade students. There was a 
positive and significant impact on the attendance rate of 6th-grade students in 2012-2013. 
Though the attendance impact was positive in 2013-2014, it was lower and not significantly 
different from zero. Suspension impact estimates were positive and significantly different from 
zero for both cohorts of 6th-grade students, though the magnitude was approximately twice as 
high for cohort II 6th- graders compared to cohort I 6th- graders. 

Overall, the suspension impact estimates were greatest for cohort I 7th graders, cohort II 6th 
graders, and cohort III 5th graders, indicating that suspensions were higher in 2013-2014 than in 
previous years. This is consistent with reports from Kauffman School teachers and staff that 
students struggled with the discipline system throughout the 2013-2014 school year, leading to 
efforts such as increased professional development sessions on the discipline system, to ensure 
that the Kauffman School’s disciplinary system was consistently implemented across all teachers 
and staff. 


A. 10 
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Table A.6. Impacts of the Kauffman School on attendance and suspensions 
for 5th-grade students between 2011-2012 and 2013-2014 



2011-2012 cohort 1 
5th-grade students 

2012-2013 cohort II 
5th-grade students 

2013-2014 cohort 

III 5th-grade 
students 

Attendance rate (%) 

-0.83 (0.48) 

0.84 (0.45) 

0.57 (0.39) 

Probability of being suspended (%) 

13.4* (5.2) 

1.5 (4.5) 

28.0** (3.6) 

Probability of in-school suspension (%) 

0.3 (3.9) 

-1.4 (1.6) 

29.1** (3.4) 

Probability of out-of-school suspension 
(%) 

14.2** (4.7) 

3.1 (4.4) 

19.4** (4.0) 

Sample size 

677 

617 

948 


Note: The suspension results are marginal effects from logit models in which the outcome variable is an indicator 

for receiving a suspension during the year. Standard errors are displayed in parentheses beside each 
impact estimate. The sample size represents the total number of Kauffman students and matched 
comparison students entering each analysis. One asterisk (*) indicates results are significantly different 
from zero at the 5 percent level. Two asterisks (**) indicate results are significantly different from zero at the 
1 percent level. 


Table A.7. Impacts of the Kauffman School on attendance and suspensions 
for 6th-grade students between 2012-2013 and 2013-2014 



2012-2013 cohort 1 6th- 
grade students 

2013-2014 cohort II 6th- 
grade students 

Attendance rate (%) 

0.89* (0.44) 

0.51 (0.46) 

Probability of being suspended (%) 

12.1* (5.1) 

24.9** (5.2) 

Probability of in-school suspension (%) 

-1.1 (3.8) 

20.0** (5.1) 

Probability of out-of-school suspension (%) 

14.0** (5.1) 

20.9** (5.6) 

Sample size 

596 

585 


Note: The suspension results are marginal effects from logit models in which the outcome variable is an indicator 

for receiving a suspension during the year. Standard errors are displayed in parentheses beside each 
impact estimate. The sample size represents the total number of Kauffman students and matched 
comparison students entering each analysis. One asterisk (*) indicates results are significantly different 
from zero at the 5 percent level. Two asterisks (**) indicate results are significantly different from zero at the 
1 percent level. 
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